Measuring Program Behavior Similarity

D esigners of microarchitectures for generalpurpose microprocessors once based their design decisions on experts’ intuition and rules of thumb. Since the mid-1980s, however, microarchitecture research has become a systematic process that uses simulation tools extensively. Although architectural simulators model microarchitectures at a high abstraction level, the increasing complexity of both the microarchitectures and the applications that run on them make these simulators very time-consuming. Simulators must execute huge numbers of instructions to create a workload representative of real applications. The Standard Performance Evaluation Corporation’s (SPEC) CPU2000 benchmark suite, for example, has many more dynamic instructions than CPU95, which it replaced. Although real hardware evaluations benefit from this increase, using architectural simulators for such large numbers of instructions becomes infeasible. The dynamic instruction count of the SPEC2000 benchmark parser with reference input is about 500 billion instructions, or three weeks of simulation at 300,000 instructions per second. Including the benchmarks that must be run for a huge number of design points creates an unreasonably long simulation time, stretching the time to market. Running the simulations in parallel results in a huge equipment cost. To solve this problem, we can use reduced input sets instead of reference input sets. The ideal reduced input set has a limited dynamic instruction count but produces program behavior comparable to the reference input set behavior. MinneSPEC collects a number of reduced input sets for some CPU2000 benchmarks. It proposes three reduced inputs: smred for short simulations, mdred for mediumlength simulations, and lgred for full-length, reportable simulations. Although a number of techniques—such as truncating or modifying the inputs—can derive these reduced input sets from the reference inputs, it is unclear whether these reduced input sets will produce behavior similar to a program using a reference input set. We have developed a methodology that reliably quantifies program behavior similarity. As such, we can validate MinneSPEC—that is, we can verify whether the reduced input sets result in program behavior similar to the reference inputs. To overcome the shortcomings of previous work, our methodology uses metrics that are closely related to performance. We also use statistical data analysis techniques to calculate the similarity in program behavior based on uncorrelated workload characteristics.